56 research outputs found
A critical analysis of self-supervision, or what we can learn from a single image
We look critically at popular self-supervision techniques for learning deep
convolutional neural networks without manual labels. We show that three
different and representative methods, BiGAN, RotNet and DeepCluster, can learn
the first few layers of a convolutional network from a single image as well as
using millions of images and manual labels, provided that strong data
augmentation is used. However, for deeper layers the gap with manual
supervision cannot be closed even if millions of unlabelled images are used for
training. We conclude that: (1) the weights of the early layers of deep
networks contain limited information about the statistics of natural images,
that (2) such low-level statistics can be learned through self-supervision just
as well as through strong supervision, and that (3) the low-level statistics
can be captured via synthetic transformations instead of using a large image
dataset.Comment: Accepted paper at the International Conference on Learning
Representations (ICLR) 202
Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probing
Self-supervised visual representation learning has recently attracted
significant research interest. While a common way to evaluate self-supervised
representations is through transfer to various downstream tasks, we instead
investigate the problem of measuring their interpretability, i.e. understanding
the semantics encoded in raw representations. We formulate the latter as
estimating the mutual information between the representation and a space of
manually labelled concepts. To quantify this we introduce a decoding
bottleneck: information must be captured by simple predictors, mapping concepts
to clusters in representation space. This approach, which we call reverse
linear probing, provides a single number sensitive to the semanticity of the
representation. This measure is also able to detect when the representation
contains combinations of concepts (e.g., "red apple") instead of just
individual attributes ("red" and "apple" independently). Finally, we propose to
use supervised classifiers to automatically label large datasets in order to
enrich the space of concepts used for probing. We use our method to evaluate a
large number of self-supervised representations, ranking them by
interpretability, highlight the differences that emerge compared to the
standard evaluation with linear probes and discuss several qualitative
insights. Code at: {\scriptsize{\url{https://github.com/iro-cp/ssl-qrp}}}.Comment: Published at ICLR 2022. Appendix included, 26 page
Semantic Counting from Self-Collages
While recent supervised methods for reference-based object counting continue
to improve the performance on benchmark datasets, they have to rely on small
datasets due to the cost associated with manually annotating dozens of objects
in images. We propose Unsupervised Counter (UnCo), a model that can learn this
task without requiring any manual annotations. To this end, we construct
"SelfCollages", images with various pasted objects as training samples, that
provide a rich learning signal covering arbitrary object types and counts. Our
method builds on existing unsupervised representations and segmentation
techniques to successfully demonstrate the ability to count objects without
manual supervision. Our experiments show that our method not only outperforms
simple baselines and generic models such as FasterRCNN, but also matches the
performance of supervised counting models in some domains.Comment: 24 pages. Code available at
https://github.com/lukasknobel/SelfCollage
Self-Ordering Point Clouds
In this paper we address the task of finding representative subsets of points
in a 3D point cloud by means of a point-wise ordering. Only a few works have
tried to address this challenging vision problem, all with the help of hard to
obtain point and cloud labels. Different from these works, we introduce the
task of point-wise ordering in 3D point clouds through self-supervision, which
we call self-ordering. We further contribute the first end-to-end trainable
network that learns a point-wise ordering in a self-supervised fashion. It
utilizes a novel differentiable point scoring-sorting strategy and it
constructs an hierarchical contrastive scheme to obtain self-supervision
signals. We extensively ablate the method and show its scalability and superior
performance even compared to supervised ordering methods on multiple datasets
and tasks including zero-shot ordering of point clouds from unseen categories
Labelling unlabelled videos from scratch with multi-modal self-supervision
A large part of the current success of deep learning lies in the
effectiveness of data -- more precisely: labelled data. Yet, labelling a
dataset with human annotation continues to carry high costs, especially for
videos. While in the image domain, recent methods have allowed to generate
meaningful (pseudo-) labels for unlabelled datasets without supervision, this
development is missing for the video domain where learning feature
representations is the current focus. In this work, we a) show that
unsupervised labelling of a video dataset does not come for free from strong
feature encoders and b) propose a novel clustering method that allows
pseudo-labelling of a video dataset without any human annotations, by
leveraging the natural correspondence between the audio and visual modalities.
An extensive analysis shows that the resulting clusters have high semantic
overlap to ground truth human labels. We further introduce the first
benchmarking results on unsupervised labelling of common video datasets
Kinetics, Kinetics-Sound, VGG-Sound and AVE.Comment: Accepted to NeurIPS 2020. Project page:
https://www.robots.ox.ac.uk/~vgg/research/selavi, code:
https://github.com/facebookresearch/selav
Less than Few: Self-Shot Video Instance Segmentation
The goal of this paper is to bypass the need for labelled examples in
few-shot video understanding at run time. While proven effective, in many
practical video settings even labelling a few examples appears unrealistic.
This is especially true as the level of details in spatio-temporal video
understanding and with it, the complexity of annotations continues to increase.
Rather than performing few-shot learning with a human oracle to provide a few
densely labelled support videos, we propose to automatically learn to find
appropriate support videos given a query. We call this self-shot learning and
we outline a simple self-supervised learning method to generate an embedding
space well-suited for unsupervised retrieval of relevant samples. To showcase
this novel setting, we tackle, for the first time, video instance segmentation
in a self-shot (and few-shot) setting, where the goal is to segment instances
at the pixel-level across the spatial and temporal domains. We provide strong
baseline performances that utilize a novel transformer-based model and show
that self-shot learning can even surpass few-shot and can be positively
combined for further performance gains. Experiments on new benchmarks show that
our approach achieves strong performance, is competitive to oracle support in
some settings, scales to large unlabelled video collections, and can be
combined in a semi-supervised setting.Comment: 25 pages, 5 figures, 13 table
Prompt Generation Networks for Input-based Adaptation of Frozen Vision Transformers
With the introduction of the transformer architecture in computer vision,
increasing model scale has been demonstrated as a clear path to achieving
performance and robustness gains. However, with model parameter counts reaching
the billions, classical finetuning approaches are becoming increasingly
limiting and even unfeasible when models become hosted as inference APIs, as in
NLP. To this end, visual prompt learning, whereby a model is adapted by
learning additional inputs, has emerged as a potential solution for adapting
frozen and cloud-hosted models: During inference, this neither requires access
to the internals of models' forward pass function, nor requires any
post-processing. In this work, we propose the Prompt Generation Network (PGN)
that generates high performing, input-dependent prompts by sampling from an
end-to-end learned library of tokens. We further introduce the "prompt
inversion" trick, with which PGNs can be efficiently trained in a latent space
but deployed as strictly input-only prompts for inference. We show the PGN is
effective in adapting pre-trained models to various new datasets: It surpasses
previous methods by a large margin on 12/12 datasets and even outperforms
full-finetuning on 5/12, while requiring 100x less parameters.Comment: Tech report, 12 pages. Code: https://github.com/jochemloedeman/PG
Efficient Neural PDE-Solvers using Quantization Aware Training
In the past years, the application of neural networks as an alternative to
classical numerical methods to solve Partial Differential Equations has emerged
as a potential paradigm shift in this century-old mathematical field. However,
in terms of practical applicability, computational cost remains a substantial
bottleneck. Classical approaches try to mitigate this challenge by limiting the
spatial resolution on which the PDEs are defined. For neural PDE solvers, we
can do better: Here, we investigate the potential of state-of-the-art
quantization methods on reducing computational costs. We show that quantizing
the network weights and activations can successfully lower the computational
cost of inference while maintaining performance. Our results on four standard
PDE datasets and three network architectures show that quantization-aware
training works across settings and three orders of FLOPs magnitudes. Finally,
we empirically demonstrate that Pareto-optimality of computational cost vs
performance is almost always achieved only by incorporating quantization.Comment: Accepted at the ICCV 2023 Workshop on Resource Efficient Deep
Learning for Computer Visio
Emergent inequality and business cycles in a simple behavioral macroeconomic model
Standard macroeconomic models assume that households are rational in the sense that they are perfect utility maximizers and explain economic dynamics in terms of shocks that drive the economy away from the steady state. Here we build on a standard macroeconomic model in which a single rational representative household makes a savings decision of how much to consume or invest. In our model, households are myopic boundedly rational heterogeneous agents embedded in a social network. From time to time each household updates its savings rate by copying the savings rate of its neighbor with the highest consumption. If the updating time is short, the economy is stuck in a poverty trap, but for longer updating times economic output approaches its optimal value, and we observe a critical transition to an economy with irregular endogenous oscillations in economic output, resembling a business cycle. In this regime households divide into two groups: poor households with low savings rates and rich households with high savings rates. Thus, inequality and economic dynamics both occur spontaneously as a consequence of imperfect household decision-making. Adding a few “rational” agents with a fixed savings rate equal to the long-term optimum allows us to match business cycle timescales. Our work here supports an alternative program of research that substitutes utility maximization for behaviorally grounded decision-making
- …